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NNEDLearn Implementation Analysis 


Evidence-Based and Culturally Relevant 
Behavioral Health Interventions in Practice: 
Strategies and Lessons Learned from 
NNEDLearn (2011-2020) 


EXECUTIVE SUMMARY 


Despite decades of research on effective interventions, behavioral health disparities continue to negatively 
affect black, indigenous, and other communities of color. The behavioral health fields can do more to 
address these inequities by creating and providing high quality and culturally appropriate services to 
specific racial and ethnic groups while also demonstrating effectiveness in improving targeted outcomes. 
An increased attention on using evidence-based practices (EBPs) as a measure of high quality care has led 
practitioners to favor evidence-based models because they are shown to be effective through outcome 
studies. While the concept that EBPs would be the most effective intervention seems sound, in theory, the 
reality is that implementing EBPs with racial and ethnically diverse populations yields many challenges. 
These challenges can be addressed by a number of factors, including: 


1) having individuals from specific racial and ethnic groups adapt existing EBPs or develop their 
own interventions based on community-defined evidence; 

2) ensuring the inclusion of racially and ethnically diverse individuals in the outcome studies, and 
building the capacity of the behavioral health workforce to implement existing culturally 
appropriate interventions. 


Since 2011 SAMHSA’s Office of Behavioral Health Equity (OBHE) has been addressing the 
aforementioned challenges, in part, through its National Network to Eliminate Disparities in Behavioral 
Health annual training (NNEDLearn) that teaches teams of practitioners from community-based 
organizations who serve black, indigenous, and other communities of color in implementing culturally 
relevant evidence-supported behavioral health practices. To assess the impact of the training on the 
alumni, qualitative interviews were conducted with a sample of nine alumni and six trainers whose 
organization participated in NNEDLearn during the period of 2011-2020. The evidence-supported 
practices included in the analysis are listed below. 


1 Achieving Whole Health Adapted for Asian Americans, Native Hawaiians, and Pacific 
Islanders 

2 Familia Adelante Developed for Latino youth and parents/caregivers 

3 Motivational Interviewing Appropriate for culturally diverse populations 


4 Preventing Long-term Anger and Developed for black male youth 
Aggression in Youth 


5 Prime Time Sister Circles Developed for black women ages 40-75 


6 Project Venture Developed for American Indian and Alaska Native Youth 





The findings from the analysis are summarized in this report in a case study format for each of the six 
practices. Each case study highlights organizational strategies to address challenges and successfully 
implement and sustain the practices over time, particularly within the context of the communities they 
serve. This report is specific to NNEDLearn, from 2011 to 2020. 


The summary of findings across all practices 


There is an evident need for culturally relevant behavioral health practices, as reported by how 
well they were received by service recipients and how well it aligned with the organizations’ 
mission. These two factors, filling a service need and alignment with organizational priorities, helped 
gain the support of executive leaders that all respondents agreed was essential to sustaining and scaling 
the practice. Tracking outcomes was another factor that facilitated leadership buy-in and created 
opportunities for funding and partnerships. 


Having congruence between the service provider and recipient on key demographic factors greatly 
facilitates implementation. The exact demographic factors varied by program but included age, gender, 
and race and ethnicity. 


Alumni attribute successful implementation to the flexibility that the practices allowed them to 
make some degree of customization to meet the needs of specific racial and ethnic groups. The 
degree of permissible adaptations varied by program. It helped that alumni could make calculated 
adaptations due to the training, which helped ground them in the theoretical basis of the model and 
provided access to the trainer/developer. 


Alumni face varying challenges, including the need to address barriers to participation. The most 
common barriers included transportation and childcare. Solutions required partnering with other 
organizations, securing grant funding, and leveraging internal funds. 


Alumni need ongoing support as they make adaptations to ensure fidelity to the model. Alumni who 
are scaling up the practice could benefit from a train-the-trainer training to better support those efforts. In 
the current context, alumni face new challenges of having moved programs to a virtual platform and seek 
support to increase participants’ engagement. 


The findings conclude that the culturally relevant evidence-informed behavioral health practices 
offered through NNEDLearn appear to be having a positive impact among racially and ethnically 
specific communities. NNEDLearn alumni are gaining the capacity required to successfully implement 
and scale up the programs that expand the reach of services. Train-the-trainer strategies and movement 
toward virtual implementation have the potential to expand reach even further, which builds progress 
toward eliminating the behavioral health disparities for black, indigenous, and other communities of 
color. Future considerations for NNEDLearn include formalizing the ongoing technical assistance 
provided by trainers that is outside of their NNEDLearn scope of work, institute evaluations inclusive of 
alumni feedback on implementation challenges to inform ongoing improvements to the model, and 
consider supplementing training with additional resources such as on-demand video training sessions that 
could contribute to implementation fidelity. 
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1. INTRODUCTION 


The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Office of Behavioral 
Health Equity (OBHE) continues its mission to reduce disparities in behavioral health by addressing the 
need to expand the capacity of the workforce in ways that improve the quality of care for underserved 
populations. Quality of care is assessed based on empirical evidence that determines which practices are 
evidence-based, and these become the gold standard of behavioral health practices. However, employing 
evidence-based models with underserved populations, including racial and ethnically diverse groups, may 
not always yield the best outcomes. This could be due to the variability in how the practices are 
implemented, especially when the person delivering the service does not have a grounding of the cultural 
context to make the necessary cultural adaptations to the service population. Another challenge is that 
many evidence-based practices (EBPs) were tested and validated on populations that often do not match 
key characteristics of underserved groups who tend to be in low-income and of different racial and ethnic 
backgrounds. The OBHE is addressing these challenges through its National Network to Eliminate 
Disparities in Behavioral Health (NNED) and NNEDLearn. 


Overview of NNEDLearn 


The NNED’s annual training, NNEDLearn, has been offered since 2011. Each year, it provides 
community-based organizations serving black, indigenous, and other communities of color an opportunity 
to receive training in one of five to six training tracks offered that year. Organizations are selected based 
on their demonstrated readiness, fit, and commitment to training and implementation. In the most recent 
years, each organization chooses three to five team members to participate. This training supports team 
members’ capacity to implement evidence-supported and culturally appropriate mental illness and 
substance use prevention and treatment practices. The training model, based on implementation science, 
consists of four critical components to support implementation as illustrated in the diagram below. 


PREPARE LEARN IMPLEMENT SUSTAIN 
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The aim of NNEDLearn is to provide the teams with the competency to implement and sustain the 
program or practice within their community. The OBHE estimates that between 2011-2020, there were 
over 1,100 individuals trained representing over 200 organizations. During this period, 29 trainers 
facilitated trainings for 17 different interventions. 


Purpose and Objectives of the Assessment 


To gain a better understanding of the longer-term outcomes of NNEDLearn, a systematic qualitative 
assessment of effective strategies and lessons learned from implementation of NNED’s evidence-based 
and evidence-informed practices was conducted. The findings highlight community-based organizations’ 
tactics in overcoming the inherent challenges of instituting new practices as well as the strategies used to 





sustain and scale up practices over time, particularly in the context of implementation within their racial 
and ethnic-specific communities. The findings are meant to highlight effective strategies in implementing 
EBPs, leading to improvements in the quality and wider adoption of effective behavioral health care 
models in underserved communities. 


The NNEDLearn training tracks selected for this analysis include the five most frequently offered as of 
2020 and an additional one that was last offered in 2013. The information for the analysis was gathered 
through individual interviews with each of the primary trainers of the selected training tracks (n=6) and 
one to two staff members from past participating organizations (alumni; n=9). Interview protocols are 
included in Appendix A. The sample of alumni was based on the trainers’ recommendation of 
organizations and the respective staff member(s) who have been successful in sustaining or scaling the 
practice and/or have applied innovative implementation strategies with racial and ethnic groups. The table 
below illustrates the breakdown of the sample. 





NNEDLearn Training Track (years offered) a. e ret strata fees eae 
| Achieving Whole Health (2013-2015, 2019-2020) | 1 | 2 | 3 
Familia Adelante (2015-2020) 1 1 2 
Motivational Interviewing (2011-2020) 1 1 2 
Preventing Long-term Anger and Aggression in Youth 1 2 3 
(2014-2020) 
Project Venture (2015-2020) 2 3 
Prime Time Sister Circles (2013) 1 1 2 
TOTAL 6 9 15 


Interview data were analyzed across all interviewees and within each training track. This report presents 
these findings in a case study format for each training track in section two and includes a summary 
section inclusive of implications and future consideration in sections three and four. The information 
included in this report provides a snapshot of strategies and lessons learned from NNEDLearn offered 
between 2011 and 2020, and is not a full comprehensive assessment across all participants and years. 


2. NNEDLEARN INTERVENTION CASE STUDIES 


Achieving Whole Health: Balancing Body, Mind, and Spirit 
THE PROGRAM 


Achieving Whole Health (AWH) is a curriculum-based, interactive, and culturally relevant program that 
focuses on an individual’s health and mental health. It employs Wellness Coaches to provide individualized 
support to community members in making healthy choices on issues such as stress management, 
improved diet and exercise, self-care skills, and building resilience through group sessions and one-on-one 
peer support. The curriculum is based on the Peer Support Whole Health & Resiliency and the Peer 
Specialist Core Recovery Curriculum developed by the Appalachian Consulting Group; adapted in 
consultation with them by Dr. DJ Ida. Program participants develop goals to meet their specific health 
needs, while helping and receiving help from others through peer support activities. AWH training track was 
offered at NNEDLearn between 2013 and 2015 and again from 2019 to 2020. 


THE INTENDED AUDIENCE 


Asian Americans, Native Hawaiians and Pacific Islanders (AANHPI) of all ages 





THE TRAINER 


DJ Ida, Ph.D., Executive Director of the National Asian American Pacific Islander Mental Health Association 
(NAAPIMHA) 


IMPLEMENTATION 


The program can be implemented in behavioral health, health, senior, student, or other community-based 
organizations focused on improving the wellbeing of AANHPIs. The program was designed to be 
portable—it doesn’t require any audio/visual equipment—and it can be implemented in various settings. An 
organizational team focused on implementation is the best approach, and it is more effective if at least one 
team member is fluent in the language of the population of focus. 


NNEDLEARN ALUMNI 


1) Asian Women for Health (AWFH), based in Boston, MA participated in NNEDLearn in 2016. Itis a 
community-based network that is peer-led and focused on advancing Asian women’s health and 
wellness through education, advocacy, and support (asianwomenforhealth.org). 





2) Asian Americans for Community Involvement (AACI), based in San Jose, CA participated in 
NNEDLearn in 2020. It is a community-based advocacy and service organization focused on 
strengthening the resilience of community members by improving their health, mental health, and 


well-being (aaci.orq). 


Key Strategies to Implementation & Sustainability 


Organizational factors 


Alumni noted how well the program fit within the context of their organizations and its mission, 
and this fit enabled them to sustain the program. For example, AACI adopted elements of the 
model in all of their services; the weaving of these elements across their services helped break 
down silos and unify the way they provide services across their programs. This was possible 
because they had buy-in from the executive leadership, and it matched their organizational vision, 
making the program easy to adopt organization-wide. The organization-wide adoption also meant 
multiple funding streams, which contributed to the program’s sustainability. 


Alumni noted the importance of selecting the right staff team to attend the training. They 
recommend selecting individuals from a variety of positions in the organization’s hierarchy, most 
importantly to include those at the top levels and those overseeing the implementation. 


In organizations that rely on volunteers for delivery of services, staff stressed the importance of 
paying people to implement AWH to keep them vested in the program’s continuity and 
sustainability. Finding the resources for a permanent paid position should be the organization’s 
goal. 


Community-based organizations tend to have better success at implementation than coalitions or 
associations because they have dedicated space and paid staff to run the program. They may also 
have a better understanding of the community’s needs because they are more accountable to the 
community compared to a coalition or association which are driven by their members or member 
organizations. 


Cultural factors 


Alumni noted that the program was a good fit with their communities. They described it as culturally 
sensitive, humble, and insightful. In particular, mental health is framed and discussed as a holistic 
approach to body, mind, and spirit, and not as a medical issue. The latter way of explaining mental health 
is not helpful for many AANHPI communities because it can be a taboo subject. 





Adaptations 


The program was adapted for AANHPIs, which is a broad umbrella that consists of many different 
cultures. That is why the program has flexibility built in, and some degree of adaptation is encouraged to 
tailor it to the needs of a specific audience. 


e Jtis important to adapt examples of holistic health to the cultural context so that they are relatable 
to the audience and is in line with cultural customs. For example, when talking about healthy 
eating, it is important to discuss food options that make sense for the participants. 


e AWFH added modules specific to Asian women focused on body image, communication, 
relationships, boundaries, and self-care. They also adapted it for teens by adding sessions on 
social media and adapting the role-plays and activities tailored to their situations. 


Implementation Challenges & Solutions 


Organizational factors 


Alumni noted the typical organizational challenges such as staff turnover, (especially at management 
levels and among those who were trained), resources (e.g., staff time, money for materials), sustainable 
funding sources, and physical spaces for in-person groups. In the current COVID-19 environment with 
virtual sessions, many participants struggle with access to or knowledge of the technology required to 
participate in online groups. 


e Inresponse to funding challenges, AWFH is exploring 1) partnerships with universities to offer 
service learning opportunities to students for serving as co-facilitators, and 2) pitching it to 
corporations by charging to train their employees to be facilitators so they can institute the 
program at their workplace and potentially also offer it to community groups. 


Language 

Alumni noted a challenge in meeting the language needs of their participants, particularly within 
multiethnic, multilingual groups. It is ideal for the program facilitator to be fluent in the languages of the 
participants, but often multiple languages are necessary especially in the AANHPI population. Since the 
program does not employ audio/visual or many written materials, it can more easily be implemented in a 
different language if the facilitator is bilingual. The curriculum is only available in English, which can 
also be challenging for facilitators who are not fluent in English. 


Additional Accomplishments 


e Some program participants were so dedicated to the AWH program, that they were able to 
volunteer and gain employment at AWFH to sustain the program. 


e AACTI applied a whole health approach internally to their own staff. They worked with their 
human resources department to establish a mindfulness room and initiatives such as healthy 
eating challenges, yoga, and art activities. 


e AACTI’s director of wellness was recognized by the President of the Board of Supervisors. 





Familia Adelante: Multi-risk Reduction Behavioral Health 
Prevention for Latino/Hispanic Youth and Families 


THE PROGRAM 


Familia Adelante (FA) is a psycho-educational, curriculum-based, culturally relevant prevention program 
focused on helping Latino families manage negative behavioral outcomes associated with stress exposure. 
The program addresses family and peer communication, positive school bonding, substance use 
prevention, and sexual health. The program was developed by Dr. Richard Cervantes and is based on his 
research to provide a multi-risk prevention program with a family development model for youth with early 
signs of behavioral or emotional distress. The FA training track was offered at NNEDLearn from 2015 to 
2020. 


THE INTENDED AUDIENCE 


Latino youth between the ages of 11-14 with risk factors for behavioral or emotional problems and their 
parent(s)/caregiver(s). 


THE TRAINER AND DEVELOPER 


Richard Cervantes, Ph.D., President/CEO of Behavioral Assessment, Inc. 


IMPLEMENTATION 


The program is intended to be delivered to youth and their parents/caregivers in separate but concurrent 
90-minute sessions in English or Spanish over 12 weeks in-person or 8 weeks virtually’. The program can 
be implemented by non-certified, paraprofessionals (e.g., community health workers/promotoras). The in- 
person program can be conducted with up to ten parents and ten youth. The virtual program may be more 
manageable with smaller groups of up to six parents and youth. It is not a treatment program and therefore 
not suitable for youth with severe behavioral or emotional issues that require clinical care. 


NNEDLEARN ALUM 


YMCA Youth and Family Services of Silver Spring (YMCA-SS), based in Silver Spring, MD, participated in 
NNEDLearn in 2019. The YMCA is a nonprofit organization focused on strengthening community by 
fostering the spiritual, mental, and physical wellbeing of youth, families, and communities. Youth and 
Family Services offers various programs including counseling, mentoring, and early intervention services 


(ymcadc.orgq). 


Key Strategies to Implementation & Sustainability 


Organizational factors 


e Organizations must have experience working with Latino youth and families and understand 
acculturation and migration stress in addition to the local community-level stressors that families 
face. Equally important is the enthusiasm of those implementing the program; they need to 
understand the needs of their community and be committed to helping them. 


e Organizations must have or create linkages to schools or other agencies that can identify and refer 
families. For example, the alum noted that their pre-existing relationship with the schools helped 
to identify and refer at-risk families to the program and afforded them the use of school facilities 
to conduct the sessions. The alum leveraged funds from other organizational programs to provide 
childcare for the younger children so that parents could participate. 


e The two biggest factors to sustaining the program over time have been to secure funding and the 
commitment from the leaders of the organization. Most organizations require external funding to 
cover the cost of implementation. Foundation grants may help to get the program started but are 
not enough to ensure sustainability. It is better to seek funding at the state or local levels or to 
partner with local school districts. Similarly, to sustain the program, it is necessary to have a 


' Tn response to the COVID-19 pandemic, the Familia Adelante program modules have been adapted for virtual online delivery. 





champion at the top levels of organization that can continue to allocate internal resources and 
make decisions about stakeholder partnerships. 


Cultural factors 


The program was developed by and intended for Latinos and, in that way, addresses issues such as 
acculturation, cultural pride, and discrimination through that specific cultural lens. The program’s focus 
on the family unit also aligns with Latino cultural values, thereby adding to its cultural relevance as a 
behavioral health intervention. It has also been found to be appropriate for all families regardless of their 
ethnic subgroups (e.g., Mexican, Puerto Rican, Cuban, etc.), whether they are US-born or not, or length of 
time residing in the US. The alum noted that it was important for the program facilitators be bilingual 
(English and Spanish) because many participants were more comfortable speaking Spanish. 


Adaptations 


The program allows for minor adaptations to be made to accommodate the needs of parent schedules and 
to customize some information to the local context. For example, one program had the sessions with 
youth after school but sessions with the parents on weekends due to their work schedules. In sessions that 
focus on gang prevention or child abuse, the information can be adapted to bring in the local laws and 
policing issues. It is also important to customize the examples and role-plays so that they are relatable to 
the audience. 


Implementation Challenges & Solutions 


e The organizational factors mentioned above were noted as challenges by the interviewees. 
Additionally, agencies that have a long history of established programming may be more resistant 
to adopting new program models, especially ones focused on specific ethnic groups. It was noted 
how much easier it is for organizations to continue doing business-as-usual than to adopt new and 
innovative models. 


e FA was designed to be a prevention program and therefore, may not be the best fit if a treatment 
program in needed instead. Selection of the program participants is important to avoid 
incongruence between need and fit of the model. 


e The alum noted several challenges to implementation: 


o At times, the curriculum felt too didactic, structured, and not interactive enough to keep the 
youth engaged. For example, the topic of a session would spark rich discussions and elicit 
emotions that had to be processed, and facilitators found it challenging to do it within the 
constraints of timed sections. They felt the model did not have enough relational aspects. 


o Retention and consistent attendance were major challenges. They attributed this to be, in 
part, due to the number and duration of sessions. They believed that some topics could be 
adequately covered in one instead of two sessions and to shorten sessions from 90 minutes 
to 60 minutes. The families in their programs tend to have high demands on their time due to 
work and other responsibilities, and the time commitment felt like too big for them. 


o They noted that the videos were too outdated to be relatable to the youth, especially in the 
sessions focused on gangs. 


o The program fared better in the summers when youth were not in school; attendance was 
higher compared to the program that ran during the school year. 


As a solution to the challenge of attendance and lack of relational aspects, the alum added a catered 
dinner option to families immediately preceding each session. 





Additional Accomplishments 


e Many organizations have gotten news coverage or been highlighted on their local television 
stations (both English and Spanish channels) for their implementation of FA. This has helped 
them gain public awareness about their work with Latinos in their community. Some groups have 
been recognized by their state government. For example, a group in Arizona is in their third or 
fourth year of being funded from the governor's office on family relations. 


e =The alum noted that the family focus is a unique aspect that creates a powerful space for families 
to work through their issues and strengthen bonds. They also noted that the program created 
bonds among the youth, and that they remained close friends after the program ended. 


Motivational Interviewing 
THE PRACTICE 


Motivational Interviewing (MI) is an evidence-based, person-centered practice that guides a style of 
engaging and communicating with an individual in a respectful manner that empowers them to change by 
drawing out their capacities and understandings. The MI training track was offered at NNEDLearn from 
2011 to 2020. 


THE INTENDED AUDIENCE 


MI works across a range of culturally diverse populations. 


THE TRAINER 
Jennifer Frey, Ph.D., Licensed Psychologist 


IMPLEMENTATION 


It is often used in the health and behavioral health fields to support change in a manner congruent with an 
individual’s own values and concerns. It is apt at addressing behaviors related to substance use, health 
promotion, medical adherence, and mental health issues. Effective use of MI will be dependent on the 
fidelity of practice and staff's ability to maintain a reasonable proficiency in delivery. To reach proficiency, 
staff must have the time and opportunities to participate in continued coaching, feedback, and practice. 
Insufficient training or inadequate implementation of MI puts individuals at risk of never achieving MI 
proficiency. 


NNEDLEARN ALUM 


1) The AIDS Resource Foundation for Children (ARFC), based in Newark, NJ participated in 
NNEDLearn in 2020. It is a nonprofit organization providing comprehensive family-centered 
services to individuals, families, and children affected by HIV/AIDS and other social and health- 
related barriers (aidsresource.org). 


Key Strategies to Implementation & Sustainability 


Organizational factors 


e While buy-in and support from organizational leadership is key to implementation regardless of 
the intervention, it is especially important for MI. Unlike the other models, MI is not a program 
rather it is a practice—a style of communication and mode of interacting with an individual. 
Therefore, in order to be adopted at the organizational level, it has to be understood and accepted 
by everyone, especially leaders who make decisions about organizational practices. 


e In addition to the support from organizational leadership to sustain the practice, the support from 
NNEDLearn is also instrumental. It has helped ARFC to have the materials from the training to 
educate other staff members, to have the continued support from the trainer, and to participate in 
discussion forums with other NNEDLearn participants. 


e MtIcan be used at all levels of care within an organization and does not require clinical training. 
The trainer mentioned research that showed if you just train your intake person—the first person 
people see when they come into the setting—it can make a significant difference in outcomes for 





the overall work, even without training anyone else. This research finding suggests that MI 
practice can create enough resilience in the person after the first interaction. 


Cultural factors 


MI works well across all communities and all cultures because it centers the need of the individual, and 
the MI practitioner decides how it is best applied within the context of the populations they engage. It is 
culturally relevant to all groups because it holds space for all experiences and is also affirming for 
individuals all along the gender spectrum. 


Implementation Challenges & Solutions 


e Since MI is not a widespread common practice, it can lead to a general lack of understanding 
among staff, which can hinder its application. The alum noted how challenging it can be to staff 
who come from more traditional health and mental health training, and even when MI is adopted, 
there is tendency to revert back to the familiar. 


e MI will not be a good fit for everyone, especially for individuals that tend to be reactive and 
controlling by nature. The trainer noted that it is a complex skill, hard to maintain, and requires 
constant practice and support that not everyone can commit to. 


Additional Accomplishments 


e The alum noted an increased engagement in services that they attributed to MI’s contribution to 
empowering participants to drive decisions about their treatment. For example, in one program, 
participants decided when and how they interacted with treatment staff (e.g., group versus 
individual sessions). MI is also known to strengthen the relationship between the practitioner and 
the service recipient, as noted through increased interest and engagement in treatment. 


e When MI leads to increased positive outcomes, other organizations notice and that has led to 
invitations to collaborate. MI has helped organizations become involved in large interagency 
projects at the state and community levels. 


Preventing Long Term Anger and Aggression in Youth 
THE PROGRAM 


Preventing Long Term Anger and Aggression in Youth (PLAAY) is a culturally relevant, curriculum-based 
program based on RECAST (Racial Encounter Coping Appraisal and Socialization Theory). It focuses on 
using the stress reduction benefits of physical activity and racial socialization to prepare youth to cope with 
stress and make healthier decisions in conflict situations. A key premise is that racial and gender-related 
conflicts are resolvable through stress management and can improve youth persistence and achievement 
in schooling. The program employs cognitive-behavioral strategies and group therapy intervention 
elements to build youth’s skills in mindfulness, positive reappraisal, stress management, and 
assertiveness. There are five core intervention strategies—storytelling, journaling, relaxation, debating, and 
role-playing. Physical activity can include basketball or other competitive activities that allow for youth to 
practice using the mental health coping skills. It was developed in 1999 by Dr. Howard Stevenson based on 
his research in this area. The PLAAY training track was offered at NNEDLearn from 2014 to 2020. 


THE INTENDED AUDIENCE 


The program was developed for black male youth and their parents/caregivers, but has been successfully 
implemented with youth of other races and genders. The age of the youth can range between 8 and 18 with 
ideal grouping as follows: 8-10; 11-13; 14-16; and 16-18 years old. 


THE TRAINER AND DEVELOPER 


Howard Stevenson, Ph.D., Professor of Education and Africana Studies at the Graduate School of 
Education, University of Pennsylvania 





IMPLEMENTATION 


The program can be implemented in schools, health agencies, recreation centers, faith-based centers, or 
other community-based organizations that focus on youth development or mentoring programs. The 
program includes 20, 90-minute sessions comprised of 45 minutes of physical activity and 45 minutes of 
culturally relevant group therapy. The ideal group size is between 10 and 15 youth. 


NNEDLEARN ALUMNI 


1) Binghamton School District (BSD), based in Binghamton, NY participated in NNEDLearn in 2017. 
It is the largest public school district in the central Southern Tier whose diverse population 
represents over 40 countries and 30 languages. It ranked 33 out of 690 as the most diverse 
district in the state in 2020-2021 school year (binghamtonschools.org). 

2) Center for Resilience (CFR), based in New Orleans, LA participated in NNEDLearn in 2020. The 


center partners with educators, clinicians, and medical practitioners to provide therapeutic day 
treatment programs to children whose behavioral health needs cannot be met in traditional 


schools (cfrla.org). 


Key Strategies to Implementation & Sustainability 


Organizational factors 


Alumni noted that the program was a natural fit to their existing activities. At BSD, the program 
supplemented the sports programs and addressed the conflict occurring within sports teams (e.g., 
football). At CFR, it complemented the group therapy programming. 


Alumni needed buy-in from leadership to expand the program. Having their organizational power 
brokers support the model allowed them to expand it to a larger institution or system. At BSD, the 
school superintendent became a champion and, with her support, PLAAY expanded district-wide. 
At CFR, after leadership saw the positive outcomes, she supported continuation and made 
implementation of PLAAY part of the staff position and lightened the clinician’s caseload. She is 
also in support of creating the organizational infrastructure necessary to evaluate program 
outcomes. 


Organizational leaders who understand the research theory of the program are more likely to 
advocate for instituting the program and its racial literacy principles into the larger organizational 
system. For example, at BSD the primary champion of PLAAY provided training to multiple 
sports coaches and was instrumental in expanding the program into classrooms and other after- 
school activities. Another organization, Love More Movement, integrated PLAAY into all the 
systems they work in, such as a network of elders that meet to prevent violence and in community 
vigils and tournaments. 


Demonstrating positive program outcomes was noted as instrumental in the program’s 
sustainability. BSD has ample resources but for funds to be allocated to the program, PLAAY 
implementers need to continue to track and report positive impact. CFR was not only able to 
sustain the program at the organization after demonstrating positive outcomes, but the leaders 
support its expansion into the community. 


At CFR, support from a community leader who works in city government and has connections to 
influential stakeholders such as the mayor, parks and recreation director, city council, and the 
school board, will be a key factor for expanding PLAAY. The CFR team strategically selected 
her to participate in the NNEDLearn training. 


Cultural factors 


With its focus on racial literacy, this program centers issues of race and culture. One aspect has been the 
inclusion of basketball as the physical activity because it is culturally relevant to black youth (the 
intended audience). However, in a multiracial setting where football plays a bigger part of the school 





culture, for example, it replaces basketball as the activity of choice for the program. Other cultural 
considerations are noted below. 


e The concordance between the facilitator and the youth on race/ethnicity is an important 
consideration for implementation. In cases where the youth participants are all of the same 
racial/ethnic group, it is helpful to have at least one facilitator be from the same group, especially 
in the group discussions. In one example, an alum noted that the presence of a white facilitator 
triggered an emotional response from a participant during a discussion about race. 


e It helps to have organizational leaders who understand the underlying theory of the program and 
the importance of integrating racial literacy into all of its programming. 


Adaptations 


e = Inconsultation with the developer, alumni have made adaptations from the program’s original 
intended audience to be inclusive of girls, other racial/ethnic groups, and wider age ranges, 
including those as young as seven. 


e BSD added a mentoring component where high school youth mentor middle and elementary 
school youth as part of the program, which was modeled after their sports program structure. 


e Alumni have substituted basketball for other athletic or interactive activities (e.g., football, board 
games). When the program moved to a virtual platform in the wake of the COVID-19 pandemic, 
that component was replaced by video games. The developer understands the need for 
substitutions of the sport but notes that the key to fidelity is engaging in an activity that is 
competitive to create the kind of stress and conflict necessary to practice coping skills. 


Implementation Challenges & Solutions 


e BSD noted logistical challenges in transporting youth to and from the program, having to 
coordinate with others for use of school facilities, and competing with the youth’s other after- 
school commitments. CFR also noted logistical challenges in having enough staff trained and able 
to devote time to implementation and, at times, not having large enough groups to have separate 
sessions for boys and girls. 


e Implementers must be prepared and open to their own personal growth. The training on racial 
literacy pushes some outside of their comfort zone and requires self-awareness in learning to 
manage difficult issues related to racial conflict. Not everyone will be up to this challenge, but 
this 1s a necessary competency for implementation. 


e The program has a parent component that organizations found challenging. Alumni noted the 
difficulties in engaging parents, especially when working with low-income families because they 
often have competing demands on their time. Increasing parent engagement would also require 
provision of support such as transportation, childcare, or meals. 


e Similar to other programs, alumni experienced a challenge in having to move to virtual 
facilitation due to the COVID-19 pandemic. Alumni noted the difficulty in maintaining the same 
level of engagement from the youth, especially for the physical activity component. As noted 
previously, CFR’s creative solution to that challenge was to use video gaming in lieu of sports. 
This adaptation is working so well, that they may continue to use this substitution even after 
COVID-19 social distancing restrictions are lifted because it is an activity that all youth enjoy and 
provides similar conflict situations to sports activities. 


Additional Accomplishments 


e The PLAAY training provided unique and incredible personal/professional growth for program 
facilitators. Some developed tight bonds with other team members who were trained, and it 





helped them cope with the social issues of 2020 (e.g., COVID-19 pandemic, civil protests). 
Alumni emphasized that the training prepares facilitators, emotionally, mentally, and spiritually. 


e The expansion of PLAAY at BSD into the classroom settings provided youth an opportunity to 
participate for academic credit that counted toward graduation. Their addition of the mentoring 
component helped high school youth develop job and interpersonal skills. Finally, their use of 
PLAAY as a diversion program, has also aided in keeping youth out of the juvenile and criminal 
justice system. 


Prime Time Sister Circles® 
THE PROGRAM 


Prime Time Sister Circles (PTSC) is a culture, gender, and age specific program focused on helping 
women make lifestyle changes to improve their health. It is curriculum-based with an interactive support 
group intervention that reduces emotional and physical health disparities and promotes positive health 
habits for mid-life black women. Based on their own research, Dr. Gayle Porter and Dr. Marilyn Gaston 
developed the program in 2003. Since then, they estimate that about 3,000 people have gone through the 
program across seven states and DC. The PTSC training track was offered at NNEDLearn in 2013. 


THE INTENDED AUDIENCE 


Black women ages 40 to 75 


THE TRAINER AND CO-DEVELOPER 
Gayle Porter, Psy.D., co-director and developer of The Gaston and Porter Health Improvement Center 


IMPLEMENTATION 


The group meets for two hours a week for 13 weeks in community-based settings (e.g., churches; libraries; 
community and health centers). 


NNEDLEARN ALUM 
1) Just Care Family Network (JCFN), Memphis TN participated in NNEDLearn in 2013. It was a 
federally funded program that ended in 2015 with a mission to change Memphis’ system by linking 
youth and families to needed services using a high fidelity wraparound approach. It established a 
comprehensive and coordinated system of care to support children with serious mental health 
needs and their families by reducing caregiver stress, preventing out-of-home placements, and 
supporting school performance. 


Key Strategies to Implementation & Sustainability 


Organizational factors 


e The organization must clearly communicate to potential participant the rationale of the program 
as well as the expectations about what the program is and is not so that they understand the 
program is not a weight loss program, for example, but one that will help them make healthier 
choices in general. 


e In most cases external funding will be necessary to cover the cost of expenses related program 
evaluation activities, the cost of training facilitators (outside of NNEDLearn), and stipends for the 
invited experts. 


e The program is more likely to be sustained if it complements existing services or fills a 
community-based established gap in services. In the case of the JCFN, the program contributed to 
the infrastructure of the system-of-care model for family support and was meeting a gap in 
services for women of that age group. They also partnered with a well-established community 
organization who wished to expand their services and that had already engaged that target 
demographic group. 
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e Evaluation is built into the curriculum and should be carried out to track outcomes that can help 
secure future funding to continue running the program. 


Cultural factors 


The program was developed specifically for black women of a certain age and therefore, tailored to fit 
their cultural context. The facilitators should be of the same demographic group, as well as the invited 
experts to the extent possible. Other considerations include: 


e For organizations that serve a wide and diverse clientele, they must support the program’s focus 
on the intended recipients—black mid-life women. They must be prepared to justify why the 
program is focused on one racial-, gender-, and age-specific group based on the research and 
theoretical foundation of the program. 


e Location is an important consideration; the circles should be carried out in places where black 
women feel safe and most accessible to them. In many cases, this means church facilities or 
public housing community rooms. 


Adaptations 


The program model allows for some customization to meet the needs of the community. The developer 
strongly recommends that organizations conduct focus groups prior to implementation to assess the needs 
by engaging a diverse group of black women in terms of ages, education, income, and professions. Ifa 
need is found to focus on a specific subgroup, if can be tailored to fit their specific needs within the 
context of the model. For example there have been organizations who have run circles for women 
recovering from opioid addiction, for example, and in those cases the stressors that are discussed are 
specific to the ones they face and the invited expert should be one who works with that population. 


Implementation Challenges & Solutions 


e In addition to potential challenges mentioned above about justifying the program’s focus on the 
target demographic group, there may be similar challenges from community members who may 
feel excluded from participating. In those cases, it speaks to a potential need for similar services 
to a wider audience and there may be other programs or services that can be offered in addition to 
PTSC. 


e Another key challenge has been facilitators wanting to adapt the program in ways that would 
compromise fidelity to the original model, such as having men as facilitators. The alum suggested 
having clearer communication from the trainer about the key threats to fidelity. 


Additional Accomplishments 


e For organizations, there have been instances in which the success of PTSC paved the way for 
additional grant funding from the funder who invested in its implementation. 


e Inthe case of JCFN, they felt the successful implementation of PTSC contributed to the receipt of 
SAMHSA’s ECHO award. 


e For facilitators, it provided an opportunity for self-reflection and self-improvement and a network 
of peer support to make their own healthier lifestyle choices. 


e For program participants, it empowered them beyond the health aspect. For example JCFN noted 
that some women felt more confident to speak in public and become advocates for other causes; it 
inspired one woman to pursue a path to become a family support specialist. In another example, 
women continued to meet after the program ended in a similar way to continue to support each 
other and work together on shared issues. 
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Project Venture: Positive Youth Development for American 
Indian and Alaska Native Youth 


THE PROGRAM 


Project Venture (PV) is an evidence-based, culturally relevant, outdoors experiential youth development 
program aimed at increasing the resilience of American Indian/Alaska Native youth. The program has a 
curriculum that involves in-school, after school, and summer camp programming including community- 
oriented service learning, outdoor adventure-based experiential activities, and classroom-based problem- 
solving and skill-building activities. The program relies on American Indian traditional values to help youth 
develop positive self-concept, effective social skills, community service ethic, internal locus of control, and 
increased decision-making and problem-solving skills. The program was developed in 1990 by McClellan 
Hall and his National Indian Youth Leadership Project. It has since been implemented in over 70 
organizations across 25 states in the U.S. and nine provinces in Canada. The PV training track was offered 
at NNEDLearn from 2015 to 2020. 


THE INTENDED AUDIENCE 


Middle school aged American Indian and Alaska Native Youth. It was initially designed and tested with 
adolescents in grades 5-9 in American Indian school and community settings in rural and low 
socioeconomic areas. It has also been successfully implemented with high school age youth and with other 
racial and ethnic groups. 


THE TRAINER AND DEVELOPER 


McClellan Hall, MA, Executive Director of the National Indian Youth Leadership Project 


IMPLEMENTATION 


Yearlong program, in school, after school, weekends, school breaks, and summers (150-200 hours). There 
is a staff to youth ratio recommendation of 1:25 in the classroom and 1:7-15 in the community-based 
component, in addition to teachers and service staff. Implementation staff will require additional skills 
training in, first aid, CPR, wilderness first responder, and/or other technical outdoor adventure skills. 


NNEDLEARN ALUMNI 


1) Lakota Youth Development (LYD; formerly Native American Advocacy Program) is located on the 
Rosebud Sioux Tribal Reservation in Herrick, SD and participated in NNEDLearn in 2015. Itis a 
statewide, non-profit, grassroots organization dedicated to helping Native youth develop healthy 
lifestyles through connections with their culture. (lakotayouthdevelopment.org). 


2) Wabanaki Public Health (WPH), based in Bangor, ME participated in NNEDLearn in 2019. The 
Tribal Public Health Unit was formed by a partnership between the four federally recognized 
Tribes in Maine and the Maine Centers for Disease Control and Prevention. In 2011 it became the 
state’s 9th public health district and is an intergovernmental relationships between the State of 
Maine and the Tribes as Sovereign Nations and focused on the wellbeing of Tribal community 
members through connection, prevention, and collaboration on issues such as obesity, substance 
misuse and prevention, and youth engagement and empowerment (wabanakipublichealth.org). 





Key Strategies to Implementation & Sustainability 
Organizational factors 


e Implementation of the program requires more resources, partners, staffing, and planning than 
most youth programs. Therefore, alumni found it very beneficial to spend the time upfront 
researching the program and carefully considering what it would take to implement it, including 
what supplies would be necessary, what partnerships would be needed, funding requirements, and 
staff capacity. 


e For the school-based component, relationships with schools and key school personnel are 
required, including establishing a formal memorandum of understanding to implement the 
program in the classroom setting. If not already established, this relationship will take time to 
develop. Alumni noted that schools may be able to resource share by providing some of the 
equipment or other supplies including use of facilities and transportation. 
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e For the community-based component, partnerships must be formed with agencies, groups, and 
individuals that can support community service learning projects and resources for outdoor 
activities. Partners that alumni have found to be helpful include guide services to provide training 
and wilderness first responder certifications. Universities often have an adventure program that 
can rent outdoor gear and local YMCAs can be an option for facilities (e.g., climbing walls). 


e For sustainability, external grants are necessary. The trainer’s experience with grants suggests 
that funders need to invest for a minimum of five years before a program can become self- 
sufficient. It is also important to be strategic about how to frame the program. At its core, it is a 
program focused on resilience, but that does not tend to attract funders. It may better framed as a 
prevention program focused on suicide, substance use, or teen pregnancy, for example, as it has 
been demonstrated to have made positive impacts in those areas. 


Cultural factors 


The program was developed by and for American Indians and Alaska Natives and is culturally grounded 
in its use of outdoor experiential learning and service learning through a strengths-based lens. The 
program is based on values that align with indigenous culture such as learning from the natural world, 
spiritual awareness, and service to others. For tailored cultural components, it is important to have access 
to and partnerships with cultural experts and/or traditional leaders. The model allows for cultural 
adaptations to specific native groups (e.g., keeping boys and girls separate if that’s the cultural norm). 


Adaptations 


e Adaptations have been made to the language, stories used, the games/activities, the food, and the 
experiential component which must take into account the local geography. For example, many of 
the typical experiential activities were not possible with Lakota youth because of the prairie 
landscape and so, they had to build their own facility. 


e Analum created an advisory group composed of elders, youth, and spiritual leaders to determine 
the adaptations necessary. 


e PV has been successfully replicated with other racial and ethnic groups beyond the original 
intended audience because it allows for the exploration of the group’s own culture. 


Implementation Challenges & Solutions 


Organizational factors 


Alumni noted many of the expected organizational challenges but the ones that stood out the most were 
those related to funding, establishing school partnerships, and staffing. Building staff capacity requires 
training beyond that related to facilitation such as becoming certified in skills such as first aid, becoming 
a wilderness first responder and emergency medical technician, which can be intimidating to some. One 
area where alumni underestimated costs was in the outdoor equipment and gear that have to be 
maintained or replaced over time. Solutions to this challenge included: 


e Recognizing that some equipment could be rented instead of purchased. 


e Analum who built the program into their long-term strategic planning was able to create a 
wellness and recovery center that will allow them to share space and equipment with other 
services (e.g., outdoor therapy, retreats) and leverage other funding/resources. 


The part of the model that involves partnering with the school system was a challenge for LYD due to the 
nature of the school standards. To reach a solution, they worked with the developer to move some aspects 
of the in-school components into an after-school program. The materials provided at the training helped 
them with talking points to approach school and other partners. 
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Additional Accomplishments 
e The program has demonstrated positive outcomes in the prevention of suicide, substance use and 
teen pregnancy among youth participants even though the model was not designed with a specific 
focus in those areas. 


e LYD worked to get the state to recognize PV as an evidence-based program and their 
organization gained accreditation as state prevention providers. The organization’s participation 
in NNEDLearn led to publicity, which elevated their credibility and increased opportunities with 
new partners (e.g., juvenile corrections facility). 


3. SUMMARY OF STRATEGIES AND LESSONS LEARNED 
FROM NNEDLEARN PROGRAMS/PRACTICES 


Factors That Facilitated Successful Implementation and Implications 


The program/practice fills a service gap. Culturally relevant programs and practices are 
more likely to succeed if they fill a service gap or need. All interviewees (n=15) reported on the success 
of implementation in terms of it being a program or practice that was well received that led to positive 
outcomes for participants. Generally speaking, most community-based organizations do have a pulse on 
the needs of their communities and can speak anecdotally to that need. However, to more effectively 
communicate the need to others, it is best to systematically document that need. Engaging the community 
in a needs assessment also allows organizations to learn about any specific needs as they relate to the 
EBP. 





IMPLICATION 


Organizations can do a needs assessment by conducting focus 
groups to solicit input from community members and use the findings 
to justify the need to leadership or funders. 





The organization values that the program/practice is grounded in research 
theory. EBPs are based on theory and research that is important for everyone involved to understand. 
All interviewees (n=15) communicated the importance of being grounded in the program/practice 
rationale and all alumni (n=9) affirmed that this was well communicated through NNEDLearn. This 
understanding allowed them to fully recognize what is required for implementation to ensure that it aligns 
with their organization and intended outcomes. It also allows them to clearly communicate and explain it 
to others to gain buy-in, request funding, or secure partners. 





IMPLICATION 


The foundational grounding in the theory and program model helps 
implementers make appropriate adaptations that will not compromise 
the quality or fidelity to the original model. 





The program/practice aligns with organizational mission. To maximize the likelihood 
that the program or practice will succeed, its underlying theory must be aligned with organizational values 
and mission. Nearly all of the alumni (n=8) commented on how well the program/practice model fit 
within the context of their work or organizational mission. The greater the alignment, the greater chance it 
will have to be built into strategic planning, infrastructure, and expansion efforts (see AWH, PLAAY, PV 
sections for examples). The more it is built into organizational strategy and infrastructure, the less that 





individual staff turnover matters. Alignment also leads to increased support from the leaders of the 
organization. 





IMPLICATION 


Organizations benefit from taking the time up front to familiarize 
themselves with the program/practice’s underlying theory and 
implementation requirements before applying to and participating in 
NNEDLearn. 





The organization has leadership buy-in. Once organizational leaders understand how the 
program or practice aligns with their mission, they are more likely to invest in its success by securing 
funding, hiring and training staff, and creating internal systems to document and track outcomes. All 
interviewees noted this as an integral factor to program/practice success. In some cases (n=5), alumni 
organizations built the program/practice or elements of it into their organizational infrastructure (see 
AWH, PLAAY, PV sections for examples). An alum who wanted to expand the program beyond the 
organization into the community, invited a community power broker to participate in NNEDLearn with 
the intent of gaining her support in community-wide implementation efforts. 





IMPLICATION 


Organizations can be strategic when selecting members of the team 
to participate in NNEDLearn to include those in positions of power to 
support program/practice implementation. 





The program/practice demonstrates impact through evaluation. Measuring and 
tracking outcomes is an important part of implementation success and sustainability. All trainers spoke to 
the importance of evaluating outcomes and how it is built into the curriculum. Several alumni also 
discussed how they are monitoring and tracking outcomes (n=5). Demonstrating positive outcomes 
helped secure grant funding. Documenting positive impact encouraged organizational leaders to continue 
supporting implementation and lead to scaling up the program or practice. For example, an alum noted 
how leadership was wary of the program in the first round of implementation but that changed once they 
saw the positive outcomes. In another example, tracking and demonstrating positive program outcomes, 
led to expansion of PV from one program in one school to multiple schools across an entire school 
district. In a larger scale example, an alum succeeded in having PV accepted as an approved EBP by the 
state and earning accreditation as a state prevention provider, which would not have been possible without 
positive outcome data. 





IMPLICATION 


Organizations should ensure that they understand what is required 
to effectively evaluate program outcomes. All of the programs and 
practices discussed here have evaluative components built into the 
model, making it easier for organizations to track and report 
outcomes. 








The program/practice integrates cultural values. All alumni noted how well the 
programs/practices fit within the context of the communities they worked with and voiced appreciation of 
the cultural relevance. Many alumni acknowledged that there were few to no other evidence-based or 
evidence-informed models that offered the same kind of behavioral health programming specific to their 
racial or ethnic populations (n=5). Part of what makes them relevant is that the program elements build on 
familiar activities or values such as service learning, sports, framing health in holistic terms, and 
strengthening family bonds. Programs center culture as a protective factor and build on cultural pride. 





IMPLICATION 


Organizations’ positive experiences in implementing the culturally 
relevant programs with their specific racial and ethnic groups can 
contribute to the evidence-base of the model, especially for those 
that work with the same or similar groups. This is where tracking 
outcomes can be important, especially when findings can be shared 
back with the developers. 





Congruence between service provider and recipient on key demographic 
characteristics is ensured. Five of the six programs/practices identify a specific racial or ethnic 
group as the intended audience, among which four of them were adhered to by alumni (AWH, FA, PTSC, 
PV). In each of those cases, alumni noted the benefits to have facilitators be of the same demographic 
group. Alumni from AWH and FA also noted that it was beneficial to have at least one facilitator who 
spoke the native language of the group. Among the programs, PTSC was the most explicit about the 
importance in having concordance between facilitator and participants on demographic factors, but all 
interviewees discussed the issue in terms of understanding the potential impact on participants. For 
example, an alum of PLAAY relayed the experience of how during a session focused on race, a white 
male facilitator’s presence alone triggered an emotional reaction from one of the black female youth 
participants. It became important for facilitators to understand the impact of his presence during the 
conversations of race, and be strategic about how and when it was appropriate for him to engage. For FA, 
because the curriculum addresses acculturation issues with both adults and youth, it helps if the facilitator 
has lived experience, and at minimum has a deep understanding of the associated stressors. 





IMPLICATION 


Organizations need to carefully consider this issue and understand 
the potential impacts of having facilitators match or not match the 
demographic makeup of their service populations and what staffing 
adjustments they may need to make. 





The program/practice allows for minor adaptations to meet the need of 
participants, especially in recognition of the diversity within racial and ethnic 
groups. While most of the programs were developed for a specific racial or ethnic group, alumni 
recognized that the heterogeneity within those groups was an important consideration to make meaningful 
adaptations to meet the needs of their groups (n=8). For example, AWH alumni had to tailor their 
examples to be relevant to the specific ethnic foods and customs of their groups. Similarly, PV alumni 
had to customize the stories, activities, and examples to the traditions of their native groups. Some 
organizations recognized a need to create groups focused more specifically on gender and age, in addition 
to race and ethnicity. For example, an AWH alum held groups specific to Asian women and another for 
Asian teenage girls. 








IMPLICATION 


The models have some degree of flexibility that allows organizations 
to customize—at minimum—the examples to the needs of the group. 
Organizations can consider segmenting groups at the intersection of 
race and ethnicity with other demographic characteristics to 
maximize those customizations and also learn if program outcomes 


differ by the different groups. 





Facilitator/CBO Benefits 
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benefits to their own personal 
growth, self reflection, and 
gave them a framework that 
helped them better understand 
and process current events, 
particularly those focused on 
health disparities and race 
relations. 


PTSC facilitators applied 
elements of the model to their 
own healthy habits. 


An AWH alum instituted 
organizational practices 
inspired by the program. They 
built a mindfulness room for 
staff and offered healthy eating 


Recognitions 


e AWH, FA, PLAAY, and 


PV alumni have 
received recognition 
through press coverage 
and/or receiving awards 
for their implementation 
of the program. 

A PTSC alum received 
SY ANI La Ya =O (@) 
award. 

An AWH alum was 
icrerole| aly 4-10 Nace M=ierl ie 
of Supervisors. 

A PV alum received 
state accreditation as a 
provider of prevention 


Summary of By-Products, Achievements, or Additional Positive Outcomes 


Youth Outcomes 


e For the alum who 
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youth mentors 
developed job skills. 
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afforded student 
participants academic 
credit. 

Recognition of PLAAY 
as a diversion 
program, helped keep 
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challenges, yoga and art services. 
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Common Challenges and Solutions to Implementation 


Alumni noted many of the expected organizational challenges to implementation, however the two that 
came up in nearly every interview were those related to gaining the support from leadership (n=9) and 
funding (n=8). 


e Alumni understood that gaining the support of organizational leaders was critical, especially in 
cases where the program or practice was perceived as innovative or was limited to a specific 
subpopulation. What helped to address this challenge was the team members’ grounding in the 
program’s rationale, justifying the community need, and tracking outcomes. 


e As alumni related the challenges related to funding, they recognized the additional considerations 
required in the delivery to underserved communities to remove any barriers to participation (e.g., 
transportation, childcare, language). Ways in which organizations addressed these challenges was 
to work with school partners to provide bus services to the youth for after school programming 
and use foundation grant money to buy a minibus to transport youth to after-school and out-of- 
school. A FA alum leveraged internal funding from other programs to provide childcare so that 
parents could attend parent sessions. Ensuring bilingual facilitators are trained and used for 
limited English proficient populations. 





An AWH alum that relies on volunteer staff had innovative solutions to the challenge of funding 
staff positions. One idea was to partner with universities that could offer service learning credit to 
students. The other was to approach corporations/businesses and introduce AWH as a service they could 
provide to their employees and charge a fee to train staff facilitators. 


For the two programs that had a family focus, alumni faced challenges engaging the parents. They 
noted that it was particularly difficult because they work with low-income families in underserved 
communities and had to first remove any barriers to participation. For example, a FA alum used internal 
funds to purchase food and provide families a meal prior to the sessions. As an incentive they also gave 
away organizational merchandise and added an awards closing ceremony. 


For PV that requires outdoor gear, wilderness training, and use of facilities such as climbing walls, 
alumni had to find creative solutions to find what they needed while keeping costs down. One 
organization partnered with a university’s Adventure Bound Center and with state-level guiding services 
to meet these some of these needs. Another resource-shared space and equipment with other 
organizational programs. 


Outstanding Barriers and Need for Continued Support 


Fidelity to the model versus need for adaptation 


All alumni noted appreciation and respect for the research and theory behind the model. However, some 
struggled with the application, finding instances in which it became challenging to implement in practice 
(n=7). 








e Alumni would like to know upfront the key unanticipated threats to 
fidelity so they can determine whether it will be feasible to 
implement within their context. This would be in addition to a 
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they can afford to do the program with or without adaptation and 
with the likelihood of positive outcomes. 


e Alumni raised a consideration for the developers; to expand 
their research base to include outcome testing of the model 
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with different groups of people. For example, if the model trainer needed 
was originally tested with middle-income families, it * ee 


should now be tested on low-income families; or if it was 
tested on AAPI populations generally, it should also test for 
differences among subgroups (e.g., South Asians versus 
Southeast Asians). The research could also consider more 


participatory research methods, not just limiting research to How best to 


experimental models. engage 
participants 
Train the trainer virtually 


Some alumni, particularly those actively seeking to expand programming, 
expressed a desire for a train-the-trainer training to gain the capacity to train 
other staff within their own organizations as well as provide training to other 
community organizations (n=4). 


Engaging participants in virtual programming 
With programs having been moved to a virtual format in 2020 during the COVID-19 pandemic, a few 
alumni noted ongoing challenges to keep participants engaged at the same level as in-person groups 





(n=3). In some cases, it was due to the nature of the activities that required adaptations beyond what could 
have ever been anticipated by the developers. In other cases, the challenge was due to participants varying 
levels of technical skills in using video conferencing or having access to internet and computers. 


4. CONCLUSION 


The findings of the analysis highlight the multiple strategies that organizations have employed to 
implement culturally relevant services in their communities. Successful implementation was due to the 
implementer’s capacity and drive and the guidance they received through NNEDLearn. The four 
components of the training model provided the support alumni needed to employ the common strategies 
of becoming grounded in the program/practice theoretical foundation, determining how it aligned with 
their organization, gaining support from decision makers, tracking outcomes, and making the right level 
of adjustments to meet the specific needs of their participants. 


Alumni confirmed what has long been understood by NNED, program developers, and anyone working to 
address behavioral health disparities—that communities need and benefit from culturally appropriate 
evidence-supported practices. The benefit to communities is dependent on having skilled practitioners 
that can implement the model with fidelity, and skilled facilitation takes dedication, commitment, and 
experience. As organizations expand programs/practices into more facets of the systems in which they 
operate, the benefits begin to increase beyond the intended outcomes at an individual, organizational, and 
community level. Alumni noted positive relationships forged among youth participants, families, and 
among practitioners. Organizations and staff have benefited from applying the model’s principles 
internally or from receiving external recognition for the success of their implementation efforts. 


Among the alumni’s common challenges were those inherent to introducing a new program/practice to an 
organization, such as staffing, funding, and gaining buy-in from leadership—all of which NNEDLearn 
has experience with, having supported hundreds of organizations in the last 10 years. Alumni expressed 
how helpful the trainers were in helping address all of their challenges, including those related to making 
program adaptations in response to the needs of their populations. In some cases, alumni generated their 
own innovative solutions and were able to consult with the trainer. 





The overall findings suggest that community-based organizations are 
successfully overcoming the challenges and meeting the needs of 
their communities through skillful delivery of the culturally relevant 

models, and NNEDLearn is helping to build their capacity. 





Implications for Training 


Overall, across all the interviewees there was a high degree of positive experiences with NNEDLearn and 
all have benefited in some way from participation in the training. However, areas for improvement were 
also raised and both are summarized below. 


Benefits to Participation 


e Alumni appreciated the trainers and their approach to training (n=6). Alumni commented on the 
trainer’s level of commitment and support and how available they were for continued support 
beyond the NNEDLearn coaching sessions. They respected how trainers were able to deliver the 
content to individuals with varying levels of experiences, education levels, and professional 
backgrounds. 


e Alumni valued the opportunity to meet the team members from other organizations and to engage 
in the discussion forums (n=5). They found it helpful to talk through planning; get ideas from 
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what others are doing; connecting with others across the country; and enjoyed the reciprocal 
support they provided to one another. 


All trainers expressed a deep appreciation of NNED, OBHE, and/or SAMHSA and the 
opportunity to be a part of NNEDLearn. They enjoy engaging with communities and having the 
venue to expose communities nationwide to the program/practice in a way that otherwise would 
not be possible. Some have found it rewarding to invite former participants help deliver the 
training. Trainers also noted that the selection of participants from the applicant pool has 
improved over time to weed out organizations that do not have serious intentions of 
implementation. 


Future Considerations for NNEDLearn 


Provision of ongoing technical assistance (TA) for alumni may be needed beyond the structured 
coaching sessions. Many trainers noted how they have continued to be available to support 
alumni well beyond their official commitment through NNEDLearn and alumni discussed how 
valuable that continued access has been to their continued success. There may be a benefit to 
formalizing that TA by created a structure or a TA Center through NNED. 


Alumni stressed the importance of ongoing training for their organizations, especially to train 
new staff. Some expressed the benefits of having attended the program/practice training multiple 
times. Others noted that continued training will help maintain fidelity to the model over time. 


It would be valuable to evaluate the programs/practices using qualitative methods. A trainer saw a 
value in using a similar line of questioning with participants before and after their participation in 
NNEDLearn. An alum noted value in providing the trainers with systematic feedback on the 
challenges to implementation to inform ongoing improvements to the model. 


To address the language challenge, an alum suggested that trainers provide online accessible 
video recorded training modules in English that can later be dubbed or subtitled in multiple 
languages. This would allow people to review the content at their own pace and repeatedly to aid 
with fidelity and ensure that adaptations remain true to original intent. 
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APPENDIX A: INTERVIEW GUIDES 


Interview Guide for NNEDLearn Alumni 


1. 


To begin, I would like to confirm that I have the correct information for the following, [name of 
organization, website, city and state, and year of participation in NNEDLearn] 


Take a moment to reflect on the training you received for [name intervention] which included the pre- 
meeting webinars, the three-day training, and the follow-up coaching sessions. 


2. 


In applying what you learned from the training to the context of your organization, describe what it 
took for your team to initially implement [name intervention]. 


a. Describe any changes your organization made to facilitate successful implementation (e.g., 
policy, practices, staffing). If no changes were made, what were the organizational factors that 
facilitated successful implementation? 

b. Did you have to secure external funding to implement the intervention (e.g., 
local/state/federal/foundation funding)? If so, describe your team’s strategy to secure the funds. 

c. Did you engage partners? If so, describe the nature of the partnership and how it contributed to 
successful implementation of the intervention. 


Has your organization sustained implementation of [name intervention]? 


Ifso: what has made it possible for your organization to sustain the intervention (e.g., making internal 
budget allocations, obtaining external funding from foundations or state block grants) 

(Probe for organizational changes, funding, partnerships, refer to 2a—2c above) 

If not: what has impeded your organization from sustaining the intervention? 


What were the key strategies you used that guided implementation of [name intervention] with your 
population of focus? (Probe for whether strategies were a result of training) 


a. Did you have to adapt [name intervention] to make it more relevant to your participants? If so, 
i. Describe the adaptations and why you made them. 
ii. Did you consult these adaptations with the trainer? If yes, what did he/she advise? 


Describe any challenges you encountered in implementing [name intervention] within the context of 
your organization, community, or population of focus. 


For each challenge they describe, probe for: 
a. How did you overcome the challenge? 
b. Did NNEDLearn training and resources contribute to the solution? If yes, how? 


What advice would you give to a similar CBO team that is in the preliminary stages of implementing 
[name intervention] for the first time? 


Were there any unexpected outcomes or additional accomplishments that resulted from your 
participation in NNEDLearn (e.g., local/state/federal/foundation funding, organizational policy 
changes, local/state awards or recognition, media coverage, new partnerships, new initiatives)? If so, 
please describe. 


What else would you like to share about your experience with NNEDLearn and implementing [name 
intervention] that will be important for me to know? 
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Interview Guide for Trainers 


1. 


To begin, I would like to confirm that I have the correct information for the following, [name, 
affiliation, role/title, and name of the intervention]. 


What are the key implementation strategies you promote in your training and coaching sessions? 


a. From your perspective, given your experience as a trainer, what are the most important factors 
that facilitate the successful implementation of [name intervention]? 

b. What are the key strategies for implementing [name intervention] with different racial and ethnic 
groups? 


Take a moment to reflect on the coaching sessions. What can you share with us about how CBO 
teams have approached the implementation of [name intervention] within the context of their 
organizations? 


a. Did they have to make changes to organizational policy, practice, staffing, or anything else? If 
yes, describe the changes they made. 

b. Was there a need to build their internal capacity beyond what NNEDLearn provided? If so, what 
capacities did they have to build? 

c. Was it necessary to access external resources such as funding or partners (e.g., 
local/state/federal/foundation funding)? If so, describe the resources. 

d. What strategies are they using to sustain implementation and why these strategies? 


What have you learned about how CBO teams have approached implementing [name intervention] 
within the context of the people they serve? 


a. Did they need to adapt [name intervention] to make it more relevant to their population? If so, 
please describe why they had to adapt and what the adaptations were. 

b. You’ve been training for several years with NNEDLearn. Did you, and if so how did you, modify 
the in-person training and coaching calls over the years? [Only Primetime Sister Circles trained 
for just one year| 


What were the most common challenges that teams encountered during implementation? 


a. What were the specific challenges related to applying [name intervention] with the racial and 
ethnic groups they serve? 

b. What guidance did you offer them in overcoming the challenges? 

c. What stands out as the most innovative solutions CBOs have used in overcoming challenges? 


What were the unexpected outcomes or additional accomplishments from their participation in 
NNEDLearn that teams mentioned (e.g., local/state/federal/foundation funding, organizational policy 
changes, local/state awards or recognition, media coverage, new partnerships, new initiatives)? 


What else would you like to share about your experience with NNEDLearn and implementing [name 
intervention] that will be important for me to know? Was there anything in particular that surprised 
you about the teams and the training? 


What would you identify as the key benefit for teams participating in NNEDLearn? What is the key 
benefit for the trainers? [optional, if time allows] 
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Recommendations for CBO teams to interview 

Our next step is to interview NNEDLearn alumni to learn about the strategies and lessons learned of their 
successful implementation of [name intervention]. We are particularly interested in teams that succeeded 
in sustaining or scaling up the intervention and/or have applied innovative strategies in implementing 
[name intervention] with racial and ethnic groups. 


Of the CBO teams you have worked with over the years at NNEDLearn, who do you recommend that we 
interview? 


e Please provide names and, if possible, the contact information of the team lead and other team 
members. If you don’t have contact information, we can get this from OBHE. We will select 1 — 
2 people per track to interview. 
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APPENDIX B: SUMMARY OF IMPLEMENTATION STRATEGIES AND 


LESSONS LEARNED 


Factors that facilitated successful implementation and implications 


The program/practice fills a service 
gap. 


The organization values that the 
program/practice is grounded in 
research theory. 


The program/practice aligns with 
organizational mission. 


The organization has leadership 
buy-in. 


The program/practice demonstrates 
impact through evaluation. 


The program/practice integrates 
cultural values. 


Congruence between service 
provider and recipient on key 
demographic characteristics is 
ensured. 


The program/practice allows minor 
adaptations to meet the need of 
participants, especially in 
recognition of the diversity within 
racial and ethnic groups. 


Organizations can do a needs assessment by conducting focus 
groups to solicit input from community members and use the findings 
to justify the need to leadership or funders. 


This knowledge allows for appropriate adaptations to be made and for 
better communication to gain buy-in, request funding, and recruit 
partners. 


Organizations benefit from taking the time up front to familiarize 
themselves with the practice’s underlying theory and implementation 
requirements before applying to and participating in NNEDLearn. 


Organizations can be strategic when selecting members of the team to 
participate in NNEDLearn to include those in positions of power to 
support practice implementation. 


The practices have evaluative components built into the model, and 
organizations should ensure that they understand what is required to 
effectively evaluate program outcomes. Sharing outcome data with 
developers and others can help to make program improvements, lead 
to new models, or add to the current evidence-base. 


Provides organizations a relevant EBP option. When evaluated, it 
contributes to the evidence-base of the model, especially when 
findings can be shared back with the developers. 


Organizations need to carefully consider when it may be an important 
issue, and understand the potential impacts of having facilitators 
match or not match the demographic makeup of their service 
populations to make any necessary staffing adjustments. 


The practices have some degree of flexibility that allows organizations 
to adapt to the needs of the group. Organizations can consider 
segmenting groups at the intersection of race and ethnicity with other 
demographic characteristics to maximize those adaptations and also 
learn if program outcomes differ by groups. Ask trainers upfront about 
the adaptations that are not allowed. 


Common Challenges and Solutions to Implementation 


Factors that helped generate solutions to overall implementation challenges included: 


e Engaging in conversations with other alumni through NNED’s discussion forum or through connections 


they made at the training. 


e One-on-one consultations with the trainers that occurred outside of formal NNEDLearn coaching 


sessions. 
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Some specific challenges and solutions that were discussed by alumni are presented in the table below. 


Funding the cost of transporting | Partner with public schools to provide busing services 
youth Secure foundation grant money to buy a minibus 


Funding the facilitator position Partner with universities that offer service learning credit to students 
Pitch the practice to businesses as something they could provide to their 
employees and charge a fee to train staff facilitators. 


Funding the cost of Partner with university centers or with state-level guiding services 

equipment/facilities Resource-share space and equipment with other organizational programs 

Funding childcare Leveraging internal funding from other programs. 

Engaging parents Provide childcare, catered dinner prior to session, organizational 
merchandise, and add a closing awards ceremony to final session. 

Gaining support from Becoming grounded in the practice rationale, justifying the community 

organizational leadership need, and tracking outcomes. 
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